Machine vision systems are used for a growing variety of manufacturing and inspection processes in research and industry. A machine vision system enables a user to monitor subject parts and workpieces, either at stationary locations or on moving lines, and to compare acquired images of the subjects to trained patterns, stored in a memory. All machine vision systems employ a camera with an image-acquisition sensor. The sensor may be based upon a charge-coupled device (CCD) or CMOS element that breaks down an optical image into a matrix of pixels, each pixel being represented by a grayscale or color value within a given range (0-255, for example). Machine vision is used to accomplish a number of different industrial, commercial and governmental tasks. For example, machine vision can be employed to decode symbology, such as barcodes and two-dimensional data matrices. Machine vision is also commonly employed to scan for defects in a part or surface. Sophisticated search and analysis software tools, in combination with powerful processors, are provided for many of these qualitative analysis functions. More basic implementations of machine vision systems have recently become available to replace older-style opto-electric detectors (sometimes termed “electric-eyes”) that have been used for many years on production lines to sense for the gross presence or absence of an item on the line, or a major defect (e.g. a missing component) in an item. Where a gross defect or missing article is detected, the detector signals an alarm and/or stops the conveyor line. The detector may operate based upon a trigger signal, provided by another opto-electric presence sensor, or based upon the timing of the moving line, which is signaled by (for example) an encoder. The detector may also cause a reject device (a “part kicker”) to eject a given article from the line.
An implementation of such a detector, termed a vision detector is described in commonly assigned, copending U.S. Published Patent Application No. US200550275831A1 (U.S. application Ser. No. 10/865,155), entitled METHOD AND APPARATUS FOR VISUAL DETECTION AND INSPECTION OF OBJECTS, by William M. Silver, the teachings of which are expressly incorporated by reference (and other related applications thereto). The vision detector employs a low-cost, high-speed image element to capture and analyze a sequence of images at higher speeds (but typically at lower resolution) than prior art vision systems. An image in such a sequence that is captured and analyzed is called a frame. The rate at which frames are captured and analyzed, called the frame rate, is sufficiently high that a moving object is seen in multiple consecutive frames as it passes through the field of view (FOV). Since the objects moves somewhat between successive frames, it is located in multiple positions in the FOV, and therefore it is seen from multiple viewing perspectives and positions relative to the illumination. Using a principle termed dynamic image analysis, viewed objects are inspected by capturing and analyzing multiple frames for which the object is located in the field of view, and basing a result on a combination of evidence obtained from each of those frames. This method provides significant advantages over prior art machine vision systems that make decisions based on a single frame. The incorporated vision detector enables visual event detection, in which events that may occur in the field of view are identified by the system. An event can be an object passing through the field of view, and by using visual event detection the object can be detected without the need for a trigger signal.
In order to obtain images from multiple perspectives, it is desirable that an object to be detected or inspected moves no more than a small fraction of the field of view between successive frames, often no more than a few pixels. For a vision detector, it is generally desirable that the object motion be no more than about one-quarter of the FOV per frame, and in typical embodiments no more than 5% or less of the FOV. It is desirable that this be achieved not by slowing down a manufacturing process but by providing a sufficiently high frame rate. In an example system the frame rate is at least 200 frames/second, and in another example the frame rate is at least 40 times the average rate at which objects are presented to the vision detector. An exemplary vision detector system capture and analyze up to 500 frames/second. This system makes use of an ultra-sensitive imager that has far fewer pixels than prior art vision systems. The high sensitivity allows very short shutter times using very inexpensive LED illumination, which in combination with the relatively small number of pixels allows very short image capture times. The imager is interfaced to a digital signal processor (DSP) that can receive and store pixel data simultaneously with analysis operations. Using a suitable software for the DSP, the time to analyze each frame generally can be kept to within the time needed to capture the next frame. The capture and analysis methods and apparatus combine to provide the desired high frame rate. By carefully matching the capabilities of the imager, DSP, and illumination with the objectives of the invention, the exemplary system can be significantly less expensive than prior art machine vision systems.
The method of visual event detection involves capturing a sequence of frames and analyzing each frame to determine evidence that an event is occurring or has occurred. When visual event detection is used to detect objects without the need for a trigger signal, the analysis would determine evidence that an object is located in the field of view.
In an exemplary method, the evidence is in the form of a value, called an object detection weight, that indicates a level of confidence that an object is located in the field of view. The value may be a simple yes/no choice that indicates high or low confidence, a number that indicates a range of levels of confidence, or any item of information that conveys evidence. One example of such a number is a so-called fuzzy logic value, further described therein. Note that no machine can make a perfect decision from an image, and so will instead make judgments based on imperfect evidence.
When performing object detection, a test is performed for each frame to decide whether the evidence is sufficient that an object is located in the field of view. If a simple yes/no value is used, the evidence may be considered sufficient if the value is “yes.” If a number is used, sufficiency may be determined by comparing the number to a threshold. Frames where the evidence is sufficient are called active frames. Note that what constitutes sufficient evidence is ultimately defined by a human user who configures the vision detector based on an understanding of the specific application at hand. The vision detector automatically applies that definition in making its decisions.
When performing object detection, each object passing through the field of view will produce multiple active frames due to the high frame rate of the vision detector. These frames may not be strictly consecutive, however, because as the object passes through the field of view there may be some viewing perspectives, or other conditions, for which the evidence that the object is located in the field of view is not sufficient. Therefore it is desirable that detection of an object begins when an active frame is found, but does not end until a number of consecutive inactive frames are found. This number can be chosen as appropriate by a user.
Once a set of active frames has been found that may correspond to an object passing through the field of view, the system typically performs a further analysis to determine whether an object has indeed been detected. This further analysis may consider some statistics of the active frames, including the number of active frames, the sum of the object detection weights, the average object detection weight, and the like.
The above-described method of dynamic image analysis involves capturing and analyzing multiple frames to inspect an object, where “inspect” is defined as determining some information about the status of the object. In one example, the status of an object includes whether or not the object satisfies inspection criteria chosen as appropriate by a user.
Dynamic image analysis can be combined with visual event detection, so that the active frames chosen by the visual event detection method are the ones used by the dynamic image analysis method to inspect the object. In an exemplary implementation, the frames used by dynamic image analysis can be captured in response to a trigger signal.
Each such frame is analyzed to determine evidence that the object satisfies the inspection criteria. In one exemplary method, the evidence is in the form of a value, called an object pass score, which indicates a level of confidence that the object satisfies the inspection criteria. As with object detection weights, the value may be a simple yes/no choice that indicates high or low confidence, a number, such as a fuzzy logic value, that indicates a range of levels of confidence, or any item of information that conveys evidence.
The status of the object may be determined from statistics of the object pass scores, such as an average or percentile of the object pass scores. The status may also be determined by weighted statistics, such as a weighted average or weighted percentile, using the object detection weights. Weighted statistics effectively weigh evidence more heavily from frames wherein the confidence is higher that the object is actually located in the field of view for that frame.
Evidence for object detection and inspection is obtained by examining a frame for information about one or more visible features of the object. A visible feature is a portion of the object wherein the amount, pattern, or other characteristic of emitted light conveys information about the presence, identity, or status of the object. Light can be emitted by any process or combination of processes, including but not limited to reflection, transmission, or refraction of a source external or internal to the object, or directly from a source internal to the object.
One aspect of the above-described vision detector is a method for obtaining evidence, including object detection weights and object pass scores, by image analysis operations on one or more regions of interest in each frame for which the evidence is needed. In an example of this method, the image analysis operation computes a measurement based on the pixel values in the region of interest, where the measurement is responsive to some appropriate characteristic of a visible feature of the object. The measurement is converted to a logic value by a threshold operation, and the logic values obtained from the regions of interest are combined to produce the evidence for the frame. The logic values can be binary or fuzzy logic values, with the thresholds and logical combination being binary or fuzzy as appropriate.
For visual event detection, evidence that an object is located in the field of view is effectively defined by the regions of interest, measurements, thresholds, logical combinations, and other parameters, which are collectively called the configuration of the vision detector, and are chosen by a user as appropriate for a given application of the invention. Similarly, the configuration of the vision detector defines what constitutes sufficient evidence. For dynamic image analysis, evidence that an object satisfies the inspection criteria is also effectively defined by the configuration of the vision detector.
In general, the vision detector is optimally adapted to perform the function of conventional opto-electric detectors. While most of these functions could be performed by a standard machine vision system, the vision detector is less expensive and easier to set up than prior art machine vision systems, and operate at much higher speeds. These systems and methods furthermore make use of multiple perspectives of moving objects, operate without triggers, provide appropriately synchronized output signals, and provide other significant and useful capabilities that will be apparent to those skilled in the art.
The above-described vision detectors, as well as other machine vision applications, such as readers of symbology (barcodes, etc.), inspection tools and facial identification systems, are generally set up, monitored, adjusted and “debugged” using Human-machine interfaces (HMIs) that employ a graphical user interface (GUI). Once set up, the vision detector and a variety of other stand-alone machine vision systems can often be disconnected from the HMI. Nevertheless, the HMI performs a critical role in the configuration of the system and subsequent analysis of system performance and ability to detect objects and their underlying features. The HMI also allows for monitoring and fault analysis in implementations wherein the detector or other vision system records events and allows playback of the recordings. For these and other purposes, a user-friendly, fully featured and versatile HMI is highly desirable.